Characterization of Errors in Retinopathy of Prematurity Diagnosis by Ophthalmologists-in-Training in Middle-Income Countries.

PURPOSE: To characterize common errors in the diagnosis of retinopathy of prematurity (ROP) among ophthalmologistsin-training in middle-income countries. METHODS: In this prospective cohort study, 200 ophthalmologists-in-training from programs in Brazil, Mexico, and the Philippines participated. A secure web-based educational system was developed using a repository of more than 2,500 unique image sets of ROP, and a reference standard diagnosis was established by combining the clinical diagnosis and the image-based diagnosis by multiple experts. Twenty web-based cases of wide-field retinal images were presented, and ophthalmologists-in-training were asked to diagnose plus disease, zone, stage, and category for each eye. Trainees' responses were compared to the consensus reference standard diagnosis. Main outcome measures were frequency and types of diagnostic errors were analyzed. RESULTS: The error rate in the diagnosis of any category of ROP was between 48% and 59% for all countries. The error rate in identifying type 2 or pre-plus disease was 77%, with a tendency for overdiagnosis (27% underdiagnosis vs 50% overdiagnosis; mean difference: 23.4; 95% CI: 12.1 to 34.7; P = .005). Misdiagnosis of treatment-requiring ROP as type 2 ROP was most commonly associated with incorrectly identifying plus disease (plus disease error rate = 18% with correct category diagnosis vs 69% when misdiagnosed; mean difference: 51.0; 95% CI: 49.3 to 52.7; P = .003). CONCLUSIONS: Ophthalmologists-in-training from middle-income countries misdiagnosed ROP more than half of the time. Identification of plus disease was the salient factor leading to incorrect diagnosis. These findings emphasize the need for improved access to ROP education to improve competency in diagnosis among ophthalmologists-in-training in middle-income countries. [J Pediatr Ophthalmol Strabismus. 2023;60(5):344-352.].

Characterization of Errors in Retinopathy of Prematurity Diagnosis by Ophthalmologists-in-Training in the United States and Canada.

PURPOSE: To identify the prominent factors that lead to misdiagnosis of retinopathy of prematurity (ROP) by ophthalmologists-in-training in the United States and Canada. METHODS: This prospective cohort study included 32 ophthalmologists-in-training at six ophthalmology training programs in the United States and Canada. Twenty web-based cases of ROP using wide-field retinal images were presented, and ophthalmologists-in-training were asked to diagnose plus disease, zone, stage, and category for each eye. Responses were compared to a consensus reference standard diagnosis for accuracy, which was established by combining the clinical diagnosis and the image-based diagnosis by multiple experts. The types of diagnostic errors that occurred were analyzed with descriptive and chi-squared analysis. Main outcome measures were frequency of types (category, zone, stage, plus disease) of diagnostic errors; association of errors in zone, stage, and plus disease diagnosis with incorrectly identified category; and performance of ophthalmologists-in-training across postgraduate years. RESULTS: Category of ROP was misdiagnosed at a rate of 48%. Errors in classification of plus disease were most commonly associated with misdiagnosis of treatment-requiring (plus error rate = 16% when treatment-requiring was correctly diagnosed vs 81% when underdiagnosed as type 2 or pre-plus; mean difference: 64.3; 95% CI: 51.9 to 76.7; P < .001) and type 2 or pre-plus (plus error rate = 35% when type 2 or pre-plus was correctly diagnosed vs 76% when overdiagnosed as treatment-requiring; mean difference: 41.0; 95% CI: 28.4 to 53.5; P < .001) disease. The diagnostic error rate of postgraduate year (PGY)-2 trainees was significantly higher than PGY-3 trainees (PGY-2 category error rate = 61% vs PGY-3 = 35%; mean difference, 25.4; 95% CI: 17.7 to 33.0; P < .001). CONCLUSIONS: Ophthalmologists-in-training in the United States and Canada misdiagnosed ROP nearly half of the time, with incorrect identification of plus disease as a leading cause. Integration of structured learning for ROP in residency education may improve diagnostic competency. [J Pediatr Ophthalmol Strabismus. 2023;60(5):337-343.].

Evaluation of an Artificial Intelligence System for Retinopathy of Prematurity Screening in Nepal and Mongolia.

Purpose: To evaluate the performance of a deep learning (DL) algorithm for retinopathy of prematurity (ROP) screening in Nepal and Mongolia. Design: Retrospective analysis of prospectively collected clinical data. Participants: Clinical information and fundus images were obtained from infants in 2 ROP screening programs in Nepal and Mongolia. Methods: Fundus images were obtained using the Forus 3nethra neo (Forus Health) in Nepal and the RetCam Portable (Natus Medical, Inc.) in Mongolia. The overall severity of ROP was determined from the medical record using the International Classification of ROP (ICROP). The presence of plus disease was determined independently in each image using a reference standard diagnosis. The Imaging and Informatics for ROP (i-ROP) DL algorithm was trained on images from the RetCam to classify plus disease and to assign a vascular severity score (VSS) from 1 through 9. Main Outcome Measures: Area under the receiver operating characteristic curve and area under the precision-recall curve for the presence of plus disease or type 1 ROP and association between VSS and ICROP disease category. Results: The prevalence of type 1 ROP was found to be higher in Mongolia (14.0%) than in Nepal (2.2%; P < 0.001) in these data sets. In Mongolia (RetCam images), the area under the receiver operating characteristic curve for examination-level plus disease detection was 0.968, and the area under the precision-recall curve was 0.823. In Nepal (Forus images), these values were 0.999 and 0.993, respectively. The ROP VSS was associated with ICROP classification in both datasets (P < 0.001). At the population level, the median VSS was found to be higher in Mongolia (2.7; interquartile range [IQR], 1.3-5.4]) as compared with Nepal (1.9; IQR, 1.2-3.4; P < 0.001). Conclusions: These data provide preliminary evidence of the effectiveness of the i-ROP DL algorithm for ROP screening in neonatal populations in Nepal and Mongolia using multiple camera systems and are useful for consideration in future clinical implementation of artificial intelligence-based ROP screening in low- and middle-income countries.

External Validation of a Retinopathy of Prematurity Screening Model Using Artificial Intelligence in 3 Low- and Middle-Income Populations.

Importance: Retinopathy of prematurity (ROP) is a leading cause of preventable blindness that disproportionately affects children born in low- and middle-income countries (LMICs). In-person and telemedical screening examinations can reduce this risk but are challenging to implement in LMICs owing to the multitude of at-risk infants and lack of trained ophthalmologists. Objective: To implement an ROP risk model using retinal images from a single baseline examination to identify infants who will develop treatment-requiring (TR)-ROP in LMIC telemedicine programs. Design, Setting, and Participants: In this diagnostic study conducted from February 1, 2019, to June 30, 2021, retinal fundus images were collected from infants as part of an Indian ROP telemedicine screening program. An artificial intelligence (AI)-derived vascular severity score (VSS) was obtained from images from the first examination after 30 weeks' postmenstrual age. Using 5-fold cross-validation, logistic regression models were trained on 2 variables (gestational age and VSS) for prediction of TR-ROP. The model was externally validated on test data sets from India, Nepal, and Mongolia. Data were analyzed from October 20, 2021, to April 20, 2022. Main Outcomes and Measures: Primary outcome measures included sensitivity, specificity, positive predictive value, and negative predictive value for predictions of future occurrences of TR-ROP; the number of weeks before clinical diagnosis when a prediction was made; and the potential reduction in number of examinations required. Results: A total of 3760 infants (median [IQR] postmenstrual age, 37 [5] weeks; 1950 male infants [51.9%]) were included in the study. The diagnostic model had a sensitivity and specificity, respectively, for each of the data sets as follows: India, 100.0% (95% CI, 87.2%-100.0%) and 63.3% (95% CI, 59.7%-66.8%); Nepal, 100.0% (95% CI, 54.1%-100.0%) and 77.8% (95% CI, 72.9%-82.2%); and Mongolia, 100.0% (95% CI, 93.3%-100.0%) and 45.8% (95% CI, 39.7%-52.1%). With the AI model, infants with TR-ROP were identified a median (IQR) of 2.0 (0-11) weeks before TR-ROP diagnosis in India, 0.5 (0-2.0) weeks before TR-ROP diagnosis in Nepal, and 0 (0-5.0) weeks before TR-ROP diagnosis in Mongolia. If low-risk infants were never screened again, the population could be effectively screened with 45.0% (India, 664/1476), 38.4% (Nepal, 151/393), and 51.3% (Mongolia, 266/519) fewer examinations required. Conclusions and Relevance: Results of this diagnostic study suggest that there were 2 advantages to implementation of this risk model: (1) the number of examinations for low-risk infants could be reduced without missing cases of TR-ROP, and (2) high-risk infants could be identified and closely monitored before development of TR-ROP.

Variability in Plus Disease Diagnosis using Single and Serial Images.

PURPOSE: To assess changes in retinopathy of prematurity (ROP) diagnosis in single and serial retinal images. DESIGN: Cohort study. PARTICIPANTS: Cases of ROP recruited from the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) consortium evaluated by 7 graders. METHODS: Seven ophthalmologists reviewed both single and 3 consecutive serial retinal images from 15 cases with ROP, and severity was assigned as plus, preplus, or none. Imaging data were acquired during routine ROP screening from 2011 to 2015, and a reference standard diagnosis was established for each image. A secondary analysis was performed using the i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1 to 9, with 9 being the most severe disease. This score has been previously demonstrated to correlate with the International Classification of ROP. Mean plus disease severity was calculated by averaging 14 labels per image in serial and single images to decrease noise. MAIN OUTCOME MEASURES: Grading severity of ROP as defined by plus, preplus, or no ROP. RESULTS: Assessment of serial retinal images changed the grading severity for > 50% of the graders, although there was wide variability. Cohen's kappa ranged from 0.29 to 1.0, which showed a wide range of agreement from slight to perfect by each grader. Changes in the grading of serial retinal images were noted more commonly in cases of preplus disease. The mean severity in cases with a diagnosis of plus disease and no disease did not change between single and serial images. The ROP VSS demonstrated good correlation with the range of expert classifications of plus disease and overall agreement with the mode class (P = 0.001). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient, 0.89). The more aggressive graders tended to be influenced by serial images to increase the severity of their grading. The VSS also demonstrated agreement with disease progression across serial images, which progressed to preplus and plus disease. CONCLUSIONS: Clinicians demonstrated variability in ROP diagnosis when presented with both single and serial images. The use of deep learning as a quantitative assessment of plus disease has the potential to standardize ROP diagnosis and treatment.

Development of Screening Criteria for Retinopathy of Prematurity in Ulaanbaatar, Mongolia, Using a Web-based Data Management System.

PURPOSE: To describe a process for identifying birth weight (BW) and gestational age (GA) screening guidelines in Mongolia. METHODS: This was a prospective cohort study in a tertiary care hospital in Ulaanbataar, Mongolia, of 193 premature infants with GA of 36 weeks or younger and/or BW of 2,000 g or less) with regression analysis to determine associations between BW and GA and the development of retinopathy of prematurity (ROP). RESULTS: As BW and GA decreased, the relative risk of developing ROP increased. The relative risk of developing any stage of ROP in infants born at 29 weeks or younger was 2.91 (95% CI: 1.55 to 5.44; P < .001] compared to older infants. The relative risk of developing any type of ROP in infants with BW of less than 1,200 g was 2.41 (95% CI: 1.35 to 4.29; P = .003] and developing type 2 or worse ROP was 2.05 (95% CI: 0.99 to 4.25; P = .05). CONCLUSIONS: Infants in Mongolia with heavier BW and older GA who fall outside of current United States screening guidelines of GA of 30 weeks or younger and/or BW of 1,500 g or less developed clinically relevant ROP. [J Pediatr Ophthalmol Strabismus. 2020;57(5):333-339.].

Evaluation of Potential Systemic Adverse Events Related to Fluorescein Angiography in Pediatric Patients.

PURPOSE: To evaluate adverse events of fluorescein angiography (FA) in pediatric patients. DESIGN: Single-institution retrospective chart review. PARTICIPANTS: Patients 0 to 18 years of age who underwent FA between January 2010 and December 2015 at a single institution in the United States. METHODS: Pediatric patients who underwent FA by 3 surgeons were included in the study. Patients with fewer than 24 hours of documented follow-up were excluded. Significant adverse events within 24 hours of FA were evaluated. Detailed intraoperative and perioperative physiological parameters, including heart rate, blood pressure, oxygen saturation, and ventilation parameters, in inpatients undergoing simultaneous examination under anesthesia were reviewed. Peri-injection effects of FA were evaluated by 2-tailed paired t test comparison of mean 5-minute preinjection and 5-minute postinjection physiological data. MAIN OUTCOME MEASURES: Significant adverse events associated with FA. RESULTS: One hundred fifteen patients with a total of 214 FA examinations were included. No significant adverse events were associated directly with FA. Comparison of mean 5-minute preinjection and postinjection physiologic parameters in 27 patients who underwent intravenous FA during EUA did not reveal significant changes associated with FA. A significant difference was found in average patient age between inpatient (2.5 years) and outpatient (10.7 years) FA (P < 0.00001). The youngest patients who underwent successful FA were 3.8 years old in the outpatient setting and 32 weeks' postmenstrual age in the inpatient setting. Patients younger than 3.8 years accounted for most (77.6%; n = 85) inpatient FA examinations. Excluding patients with a need or likely need for laser or surgery, the reasons for inpatient FA in patients older than 3.8 years included the lack of availability of outpatient ultra-widefield FA (UWFA) and more challenging situations in patients with developmental delay. CONCLUSIONS: Fluorescein angiography was not found to be associated directly with systemic adverse events in pediatric patients in this study. Younger patients more commonly were found to require an inpatient FA, whereas older patients older than 4 years underwent outpatient UWFA.

Training of Residents and Fellows in Retinopathy of Prematurity Around the World: An International Web-Based Survey.

PURPOSE: To characterize retinopathy of prematurity (ROP) training practices in international residency and fellowship programs. METHODS: A publicly available online-based platform (http://www.SurveyMonkey.com) was used to develop a 28-question multiple-choice survey that targeted ROP screening and treatment methods. The authors solicited training programs in the Philippines, Thailand, and Taiwan. RESULTS: Programs from three countries participated in the survey, and a total of 95 responses collected from residents, fellows, and attending ophthalmologists were analyzed. A descriptive analysis demonstrated that 45 participants (47%) reported 1% to 33% of ROP screenings were performed under direct supervision of attending ophthalmologists, and 35 (37%) reported the use of formal assessments. The majority of participants (Country A: 87%, Country B: 71%, and Country C: 75%) estimated 1% to 33% of their practice was spent screening for ROP. Notably, 44 participants (46%) reported performing zero laser photocoagulation treatments for ROP during training (Country A: 65%, Country B: 38%, and Country C: 38%). CONCLUSIONS: International ophthalmology trainees perform a limited number of ROP examinations and laser interventions. ROP screenings are often unsupervised and lead to no formal evaluation by an attending ophthalmologist. Limited ROP training among ophthalmologists may lead to misdiagnosis and ultimately mismanagement of a patient. Loss of vision and exposure to unwarranted treatments are among the implications of such errors. The findings highlight the need to improve ROP training in international ophthalmology residency and fellowship programs. [J Pediatr Ophthalmol Strabismus. 2019;56(5):282-287.].

Automated Fundus Image Quality Assessment in Retinopathy of Prematurity Using Deep Convolutional Neural Networks.

PURPOSE: Accurate image-based ophthalmic diagnosis relies on fundus image clarity. This has important implications for the quality of ophthalmic diagnoses and for emerging methods such as telemedicine and computer-based image analysis. The purpose of this study was to implement a deep convolutional neural network (CNN) for automated assessment of fundus image quality in retinopathy of prematurity (ROP). DESIGN: Experimental study. PARTICIPANTS: Retinal fundus images were collected from preterm infants during routine ROP screenings. METHODS: Six thousand one hundred thirty-nine retinal fundus images were collected from 9 academic institutions. Each image was graded for quality (acceptable quality [AQ], possibly acceptable quality [PAQ], or not acceptable quality [NAQ]) by 3 independent experts. Quality was defined as the ability to assess an image confidently for the presence of ROP. Of the 6139 images, NAQ, PAQ, and AQ images represented 5.6%, 43.6%, and 50.8% of the image set, respectively. Because of low representation of NAQ images in the data set, images labeled NAQ were grouped into the PAQ category, and a binary CNN classifier was trained using 5-fold cross-validation on 4000 images. A test set of 2109 images was held out for final model evaluation. Additionally, 30 images were ranked from worst to best quality by 6 experts via pairwise comparisons, and the CNN's ability to rank quality, regardless of quality classification, was assessed. MAIN OUTCOME MEASURES: The CNN performance was evaluated using area under the receiver operating characteristic curve (AUC). A Spearman's rank correlation was calculated to evaluate the overall ability of the CNN to rank images from worst to best quality as compared with experts. RESULTS: The mean AUC for 5-fold cross-validation was 0.958 (standard deviation, 0.005) for the diagnosis of AQ versus PAQ images. The AUC was 0.965 for the test set. The Spearman's rank correlation coefficient on the set of 30 images was 0.90 as compared with the overall expert consensus ranking. CONCLUSIONS: This model accurately assessed retinal fundus image quality in a comparable manner with that of experts. This fully automated model has potential for application in clinical settings, telemedicine, and computer-based image analysis in ROP and for generalizability to other ophthalmic diseases.

Early Diagnosis and Management of Aggressive Posterior Vitreoretinopathy Presenting in Premature Neonates.

BACKGROUND AND OBJECTIVE: Aggressive posterior vitreoretinopathy (APVR) manifests with a broad area of retinal avascularity, progressive neovascularization, and/or tractional retinal detachment during the neonatal period. PATIENTS AND METHODS: A multicenter, retrospective, observational, consecutive case series study was performed to evaluate the retinal findings and structural retinal outcomes in patients treated for APVR within the first 3 months of life. RESULTS: Three premature neonates with a non-retinopathy of prematurity (ROP) APVR identified during routine ROP screening exams exhibited relatively severe, rapidly progressive retinal vascular abnormalities. Immediate laser photocoagulation of the avascular retina and vitrectomy for traction retinal detachment within several days to weeks improved or stabilized the retinal anatomy in all cases. CONCLUSIONS: This series describes clinical features in APVR in premature infants and suggests that early diagnosis and intervention may mitigate the typical aggressive course and poor prognosis of this condition. [Ophthalmic Surg Lasers Imaging Retina. 2019;50:201-207.].

Accuracy and Reliability of Eye-Based vs Quadrant-Based Diagnosis of Plus Disease in Retinopathy of Prematurity.

Importance: Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants. Objective: To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process. Design, Setting, and Participants: In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017. Interventions: Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc. Main Outcomes and Measures: Intragrader and intergrader reliability (absolute agreement and κ statistic) and accuracy compared with the reference standard diagnosis. Results: Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen κ range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss κ, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader. Conclusions and Relevance: Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

Telemedical Diagnosis of Stage 4 and Stage 5 Retinopathy of Prematurity.

PURPOSE: To determine the accuracy of image-based diagnosis for stage 4 or worse retinopathy of prematurity (ROP) disease. DESIGN: Prospective cohort study. PARTICIPANTS: We prospectively obtained data, from 8 major ROP centers, for 1220 eye examinations from 230 infants. METHODS: An ophthalmologist at each center provided a clinical diagnosis using indirect ophthalmoscopy. Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were then obtained, and these were independently read by 2 ROP experts using a web-based system for an image-based diagnosis. MAIN OUTCOME MEASURES: Sensitivity and specificity of image-based diagnosis from the ROP experts were calculated using the clinical diagnosis as the reference standard. RESULTS: Of 1220 examinations, 28 (2%) had a clinical diagnosis of stage 4 or worse. Sensitivity and specificity for stage 4 or worse disease were 75% and 99% for expert 1, and 86% and 99% for expert 2. Sensitivity and specificity for the detection of stage 5 disease were 69% and 99% for both experts. CONCLUSIONS: There are inconsistencies in the accuracy of image-based diagnosis of stage 4 and stage 5 ROP when compared with the clinical diagnosis.

Deep Learning for Image Quality Assessment of Fundus Images in Retinopathy of Prematurity.

Accurate image-based medical diagnosis relies upon adequate image quality and clarity. This has important implications for clinical diagnosis, and for emerging methods such as telemedicine and computer-based image analysis. In this study, we trained a convolutional neural network (CNN) to automatically assess the quality of retinal fundus images in a representative ophthalmic disease, retinopathy of prematurity (ROP). 6,043 wide-angle fundus images were collected from preterm infants during routine ROP screening examinations. Images were assessed by clinical experts for quality regarding ability to diagnose ROP accurately, and were labeled "acceptable" or "not acceptable." The CNN training, validation and test sets consisted of 2,770 images, 200 images, and 3,073 images, respectively. Test set accuracy was 89.1%, with area under the receiver operating curve equal to 0.964, and area under the precision-recall curve equal to 0.966. Taken together, our CNN shows promise as a useful prescreening method for telemedicine and computer-based image analysis applications. We feel this methodology is generalizable to all clinical domains involving image-based diagnosis.

Changes in Relative Position of Choroidal Versus Retinal Vessels in Preterm Infants.

Purpose: The purpose of this study was to characterize a novel finding that relative positions of choroidal and retinal vessels change over time in preterm infants and to identify factors associated with this finding using quantitative analysis. Methods: Fundus images were obtained prospectively through a retinopathy of prematurity (ROP) cohort study. Images were excluded if choroidal vessels could not be identified. Changes in relative position of characteristic choroidal landmarks with respect to retinal vessels between two time points 5 to 7 weeks apart were measured. Univariate and multivariate regression analyses were performed to identify associated factors with the amount of change. Results: The discovery and replication cohorts included 45 and 58 patients, respectively. Ninety-two of them (89%) were non-Hispanic Caucasians. Changes in relative position of choroidal versus retinal vessels were detected in all eyes of the discovery and replication cohorts (mean amount = 0.42 ± 0.12 and 0.35 ± 0.12 mm, respectively). On combined multiple regression analysis of the two cohorts, type 1 ROP, higher postmenstral age at the first time point, and shorter distance from optic disc to choroidal landmark were significantly associated with less change in relative position. Conclusions: Choroidal vessels grow anteriorly with respect to retinal vessels at posterior pole in preterm infants, suggesting relatively faster peripheral growth of choroidal versus retinal vessels. Eyes with severe ROP showed less difference in growth, which might represent alterations in choroidal development due to advanced ROP. These findings may contribute to better understanding about the physiology of choroidal development and involvement in ROP.

Assessment of a Tele-education System to Enhance Retinopathy of Prematurity Training by International Ophthalmologists-in-Training in Mexico.

PURPOSE: To evaluate a tele-education system developed to improve diagnostic competency in retinopathy of prematurity (ROP) by ophthalmologists-in-training in Mexico. DESIGN: Prospective, randomized cohort study. PARTICIPANTS: Fifty-eight ophthalmology residents and fellows from a training program in Mexico consented to participate. Twenty-nine of 58 trainees (50%) were randomized to the educational intervention (pretest, ROP tutorial, ROP educational chapters, and posttest), and 29 of 58 trainees (50%) were randomized to a control group (pretest and posttest only). METHODS: A secure web-based educational system was created using clinical cases (20 pretest, 20 posttest, and 25 training chapter-based) developed from a repository of over 2500 unique image sets of ROP. For each image set used, a reference standard ROP diagnosis was established by combining the clinical diagnosis by indirect ophthalmoscope examination and image-based diagnosis by multiple experts. Trainees were presented with image-based clinical cases of ROP during a pretest, posttest, and training chapters. MAIN OUTCOME MEASURES: The accuracy of ROP diagnosis (e.g., plus disease, zone, stage, category) was determined using sensitivity and specificity calculations from the pretest and posttest results of the educational intervention group versus control group. The unweighted kappa statistic was used to analyze the intragrader agreement for ROP diagnosis by the ophthalmologists-in-training during the pretest and posttest for both groups. RESULTS: Trainees completing the tele-education system had statistically significant improvements (P < 0.01) in the accuracy of ROP diagnosis for plus disease, zone, stage, category, and aggressive posterior ROP (AP-ROP). Compared with the control group, trainees who completed the ROP tele-education system performed better on the posttest for accurately diagnosing plus disease (67% vs. 48%; P = 0.04) and the presence of ROP (96% vs. 91%; P < 0.01). The specificity for diagnosing AP-ROP (94% vs. 78%; P < 0.01), type 2 ROP or worse (92% vs. 84%; P = 0.04), and ROP requiring treatment (89% vs. 79%; P < 0.01) was better for the trainees completing the tele-education system compared with the control group. Intragrader agreement improved for identification of plus disease, zone, stage, and category of ROP after completion of the educational intervention. CONCLUSIONS: A tele-education system for ROP education was effective in improving the diagnostic accuracy of ROP by ophthalmologists-in-training in Mexico. This system has the potential to increase competency in ROP diagnosis and management for ophthalmologists-in-training from middle-income nations.

Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability.

PURPOSE: To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP). DESIGN: We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD. PARTICIPANTS: Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units. METHODS: Expert classification of images of plus disease in ROP. MAIN OUTCOME MEASURES: Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement). RESULTS: There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0-0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2 = 0.82; and dataset B: P < 0.05 and adjusted R2 = 0.6615). CONCLUSIONS: There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.

Influence of Computer-Generated Mosaic Photographs on Retinopathy of Prematurity Diagnosis and Management.

IMPORTANCE: Telemedicine is becoming an increasingly important component of clinical care for retinopathy of prematurity (ROP), but little information exists regarding the role of mosaic photography for ROP telemedicine diagnosis. OBJECTIVE: To examine the potential effect of computer-generated mosaic photographs on the diagnosis and management of ROP. DESIGN, SETTING, AND PARTICIPANTS: In this prospective cohort study performed from July 12, 2011, through September 21, 2015, images were acquired from ROP screening at 8 academic institutions, and ROP experts interpreted 40 sets (20 sets with individual fundus photographs with ≥3 fields and 20 computer-generated mosaic photographs) of wide-angle retinal images from infants with ROP. All experts independently reviewed the 40 sets and provided a diagnosis and management plan for each set presented. MAIN OUTCOMES AND MEASURES: The primary outcome measure was the sensitivity and specificity of the ROP diagnosis by experts that was calculated using a consensus reference standard diagnosis, determined from the diagnosis of fundus photographs by 3 experienced readers in combination with the clinical diagnosis based on ophthalmoscopic examination. Mean unweighted κ statistics were used to analyze the mean intergrader agreement among experts for diagnosis of zone, stage, plus disease, and category. RESULTS: Nine ROP experts (4 women and 5 men) who have been practicing ophthalmology for a mean of 10.8 years (range, 3-24 years) consented to participate. Diagnosis by the mosaic photographs compared with diagnosis by multiple individual photographs resulted in improvements in sensitivity for diagnosis of stage 2 disease or worse (95.9% vs 88.9%; difference, 7.0; 95% CI, 3.5 to 10.5; P = .02), plus disease (85.7% vs 63.5%; difference, 22.2; 95% CI, 7.6 to 36.9; P = .02), and treatment-requiring ROP (84.4% vs 68.5%; difference, 15.9; 95% CI, 0.8 to 31.7; P = .047). With use of the κ statistic, mosaic photographs, compared with multiple individual photographs, resulted in improvements in intergrader agreement for diagnosis of plus disease or not (0.54 vs 0.40; mean κ difference, 0.14; 95% CI, 0.07 to 0.21; P = .004), stage 3 disease or worse or not (0.60 vs 0.52; mean κ difference, 0.06; 95% CI, -0.06 to 0.18; P = .04), and type 2 ROP or not (0.58 vs 0.51; mean κ difference, 0.07; 95% CI, 0.03 to 0.11; P = .04). After viewing the mosaic photographs, experts altered their choice of management in 42 of 180 responses (23.3%; 95% CI, 17.1%-29.5%). CONCLUSIONS AND RELEVANCE: Compared with multiple individual photographs, computer-generated mosaic photographs were associated with improved accuracy of image-based diagnosis for certain categories (eg, plus disease, stage 2 disease or worse, and treatment-requiring ROP) of ROP by experts. It is unclear, however, whether these findings are generalizable, and the results of this study may not be relevant to mosaic grading of other retinal vascular conditions.

Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis.

PURPOSE: To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP. DESIGN: We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases. PARTICIPANTS: Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units. METHODS: Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system. MAIN OUTCOME MEASURES: Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling. RESULTS: There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86). CONCLUSIONS: Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.